Visual tracking of peripheral devices

ABSTRACT

Techniques are disclosed for performing localization of a handheld device with respect to a wearable device. At least one sensor mounted to the handheld device, such as an inertial measurement unit (IMU), may obtain handheld data indicative of movement of the handheld device with respect to the world. An imaging device mounted to either the handheld device or the wearable device may capture a fiducial image containing a number of fiducials affixed to the other device. The number of fiducials contained in the image are determined. Based on the number of fiducials, at least one of a position and an orientation of the handheld device with respect to the wearable device are updated based on the image and the handheld data in accordance with a first operating state, a second operating state, or a third operating state.

CROSS-REFERENCES TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.16/588,830 filed Sep. 30, 2019, entitled “VISUAL TRACKING OF PERIPHERALDEVICES,” which is a continuation of International Patent ApplicationNo. PCT/US2019/021025 filed Mar. 6, 2019, entitled “VISUAL TRACKING OFPERIPHERAL DEVICES,” which claims the benefit of and priority to U.S.Provisional Patent Application No. 62/640,299 filed Mar. 8, 2018,entitled “VISUAL TRACKING OF PERIPHERAL DEVICES” and to U.S. ProvisionalPatent Application No. 62/640,009 filed Mar. 7, 2018, entitled “VISUALTRACKING OF PERIPHERAL DEVICES,” the entire disclosures of which arehereby incorporated by reference, for all purposes, as if fully setforth herein.

BACKGROUND OF THE INVENTION

Modern computing and display technologies have facilitated thedevelopment of systems for so called “virtual reality” or “augmentedreality” experiences, wherein digitally reproduced images or portionsthereof are presented to a user in a manner wherein they seem to be, ormay be perceived as, real. A virtual reality, or “VR,” scenariotypically involves presentation of digital or virtual image informationwithout transparency to other actual real-world visual input; anaugmented reality, or “AR,” scenario typically involves presentation ofdigital or virtual image information as an augmentation to visualizationof the actual world around the user.

Despite the progress made in these display technologies, there is a needin the art for improved methods, systems, and devices related toaugmented reality systems, particularly, display systems.

SUMMARY OF THE INVENTION

The present invention relates generally to localization (position,orientation, and/or distance) of a peripheral device. More particularly,embodiments of the present invention provide systems, devices, andmethods for localization of a handheld device with respect to a wearabledevice. Although portions of the present disclosure are described inreference to an augmented reality (AR) system, the disclosure isapplicable to a variety of applications.

In accordance with a first aspect of the present invention, a method ofperforming localization of a handheld device with respect to a wearabledevice is provided. The method may include obtaining, by at least onesensor mounted to the handheld device, handheld data indicative ofmovement of the handheld device with respect to the world. In someembodiments, obtaining the handheld data includes detecting, by aninertial measurement unit (IMU) mounted to the handheld device, linearaccelerations and rotational velocities of the handheld device. In someembodiments, obtaining the handheld data includes capturing, by ahandheld camera mounted to the handheld device, a world image containingone or more features surrounding the handheld device. The method mayfurther include obtaining, by a wearable camera mounted to the wearabledevice, fiducial data indicative of movement of the handheld device withrespect to the wearable device. In some embodiments, obtaining thefiducial data includes capturing, by the wearable camera, a fiducialimage containing a number of light-emitting diodes (LEDs) affixed to thehandheld device of a plurality of LEDs affixed to the handheld device.

The method may further include determining the number of LEDs containedin the fiducial image. The method may further include in response todetermining that the number of LEDs is equal to or greater than three,updating the position and the orientation of the handheld device withrespect to the wearable device based solely on the fiducial data inaccordance with a first operating state. The method may further includein response to determining that the number of LEDs is equal to one ortwo, updating the position and the orientation of the handheld devicewith respect to the wearable device based on the fiducial data and thehandheld data in accordance with a second operating state. The methodmay further include in response to determining that the number of LEDsis equal to zero, updating the position and the orientation of thehandheld device with respect to the wearable device based solely on thehandheld data in accordance with a third operating state.

In accordance with a second aspect of the present invention, a method ofperforming localization of a handheld device with respect to a wearabledevice. The method may include obtaining, by at least one sensor mountedto the handheld device, handheld data indicative of movement of thehandheld device with respect to the world. The method may also includeobtaining, by an imaging device mounted to a first device, fiducial dataindicative of movement of the handheld device with respect to thewearable device. In some embodiments, the first device is either thehandheld device or the wearable device. In some embodiments, obtainingthe fiducial data includes capturing, by the imaging device, a fiducialimage containing a number of fiducials affixed to a second devicedifferent than the first device. In some embodiments, the second deviceis either the handheld device or the wearable device. The method mayfurther include determining the number of fiducials contained in thefiducial image. The method may further include based on the number offiducials contained in the fiducial image, updating a position and anorientation of the handheld device with respect to the wearable devicebased on the fiducial data and the handheld data in accordance with afirst operating state or a second operating state.

In some embodiments, obtaining the handheld data includes detecting, byan IMU mounted to the handheld device, rotational movement of thehandheld device. In some embodiments, the imaging device is mounted tothe handheld device and a plurality of fiducials including the number offiducials are affixed to the wearable device. In some embodiments, theimaging device is mounted to the wearable device and a plurality offiducials including the number of fiducials are affixed to the handhelddevice. In some embodiments, the imaging device is mounted to thehandheld device and a plurality of fiducials including the number offiducials are affixed to the wearable device. In some embodiments,obtaining the handheld data includes capturing, by a second handheldimaging device mounted to the handheld device, a world image containingone or more features surrounding the handheld device. In someembodiments, the imaging device is mounted to the wearable device and asingle fiducial including the number of fiducials is affixed to thehandheld device. In such embodiments, obtaining the handheld dataincludes capturing, by a second handheld imaging device mounted to thehandheld device, a world image containing one or more featuressurrounding the handheld device.

In some embodiments, the imaging device is mounted to the wearabledevice and a plurality of fiducials including the number of fiducialsare affixed to the handheld device. In such embodiments, obtaining thehandheld data includes capturing, by a second handheld imaging devicemounted to the handheld device, a world image containing one or morefeatures surrounding the handheld device. The method may further includein response to determining that the number of fiducials is equal to orgreater than three, updating the position and the orientation of thehandheld device with respect to the wearable device based on thefiducial data in accordance with a first operating state. The method mayfurther include in response to determining that the number of fiducialsis equal to one or two, updating the position and the orientation of thehandheld device with respect to the wearable device based on thefiducial data and the handheld data in accordance with a secondoperating state. The method may further include in response todetermining that the number of fiducials is equal to zero, updating theposition and the orientation of the handheld device with respect to thewearable device based on the handheld data in accordance with a thirdoperating state. In some embodiments, the position and the orientationof the handheld device with respect to the wearable device is updatedbased solely on the fiducial data in accordance with the first operatingstate. In some embodiments, the position and the orientation of thehandheld device with respect to the wearable device is updated basedsolely on the handheld data in accordance with the third operatingstate.

In accordance with a third aspect of the present invention, a system forperforming localization of a handheld device with respect to a wearabledevice is provided. The system may include the wearable device. Thesystem may also include the handheld device. The system may furtherinclude one or more processors communicatively coupled to the wearabledevice and the handheld device. In some embodiments, the one or moreprocessors are configured to perform operations including obtaining, byat least one sensor mounted to the handheld device, handheld dataindicative of movement of the handheld device with respect to the world.The operations may also include obtaining, by an imaging device mountedto a first device, fiducial data indicative of movement of the handhelddevice with respect to the wearable device. In some embodiments, thefirst device is either the handheld device or the wearable device. Insome embodiments, obtaining the fiducial data includes capturing, by theimaging device, a fiducial image containing a number of fiducialsaffixed to a second device different than the first device. In someembodiments, the second device is either the handheld device or thewearable device. The operations may further include determining thenumber of fiducials contained in the fiducial image. The operations mayfurther include based on the number of fiducials contained in thefiducial image, updating a position and an orientation of the handhelddevice with respect to the wearable device based on the fiducial dataand the handheld data in accordance with a first operating state or asecond operating state.

In some embodiments, obtaining the handheld data includes detecting, byan IMU mounted to the handheld device, rotational movement of thehandheld device. In some embodiments, the imaging device is mounted tothe handheld device and a plurality of fiducials including the number offiducials are affixed to the wearable device. In some embodiments, theimaging device is mounted to the wearable device and a plurality offiducials including the number of fiducials are affixed to the handhelddevice. In some embodiments, the imaging device is mounted to thehandheld device and a plurality of fiducials including the number offiducials are affixed to the wearable device. In such embodiments,obtaining the handheld data includes capturing, by a second handheldimaging device mounted to the handheld device, a world image containingone or more features surrounding the handheld device. In someembodiments, the imaging device is mounted to the wearable device and asingle fiducial including the number of fiducials is affixed to thehandheld device. In such embodiments, obtaining the handheld dataincludes capturing, by a second handheld imaging device mounted to thehandheld device, a world image containing one or more featuressurrounding the handheld device. In some embodiments, the imaging deviceis mounted to the wearable device and a plurality of fiducials includingthe number of fiducials are affixed to the handheld device. In suchembodiments, obtaining the handheld data includes capturing, by a secondhandheld imaging device mounted to the handheld device, a world imagecontaining one or more features surrounding the handheld device.

In some embodiments, the operations further include in response todetermining that the number of fiducials is equal to or greater thanthree, updating the position and the orientation of the handheld devicewith respect to the wearable device based on the fiducial data inaccordance with a first operating state. In some embodiments, theoperations further include in response to determining that the number offiducials is equal to one or two, updating the position and theorientation of the handheld device with respect to the wearable devicebased on the fiducial data and the handheld data in accordance with asecond operating state. In some embodiments, the operations furtherinclude in response to determining that the number of fiducials is equalto zero, updating the position and the orientation of the handhelddevice with respect to the wearable device based on the handheld data inaccordance with a third operating state.

Numerous benefits are achieved by way of the present invention overconventional techniques. For example, embodiments of the presentinvention offer higher accuracy localization of a handheld device thanconventional techniques, such as electromagnetic tracking systems whichemploy a series of magnetic coils. Embodiments of the present inventionmay also make use of hardware already being utilized by an AR system,such as the front-facing or side-facing world cameras equipped on thehead set. Embodiments may extend beyond AR systems and into anyapplication where localization of one device with respect to anotherdevice is important. Other benefits of the present invention will bereadily apparent to those skilled in the art.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates an augmented reality (AR) scene as viewed through awearable AR device according to an embodiment described herein.

FIG. 2 illustrates various possible components of an AR system.

FIG. 3 illustrates an example of how a visual tracking system may beincorporated with an AR system having a wearable device and a handhelddevice.

FIG. 4 illustrates a diagram of a localization task.

FIG. 5 illustrates an example configuration of an AR system.

FIG. 6 illustrates a method of performing localization using the exampleconfiguration illustrated in FIG. 5.

FIG. 7 illustrates an example configuration of an AR system.

FIG. 8 illustrates a method of performing localization using the exampleconfiguration illustrated in FIG. 7.

FIG. 9 illustrates an example configuration of an AR system.

FIG. 10 illustrates a method of performing localization using theexample configuration illustrated in FIG. 9.

FIG. 11A illustrates an example configuration of an AR system.

FIG. 11B illustrates an example configuration of an AR system.

FIG. 12 illustrates a method of performing localization using theexample configurations illustrated in FIGS. 11A and 11B.

FIG. 13 illustrates a method of performing localization using any of thepreviously illustrated example configurations.

FIG. 14 illustrates a simplified computer system according to someembodiments described herein.

DETAILED DESCRIPTION OF SPECIFIC EMBODIMENTS

In conventional virtual reality (VR) or augmented reality (AR) systems,six degrees of freedom tracking of a peripheral device is achieved byincorporating a series of electromagnetic sensors and emitters that arestrategically placed on the user's AR headset, belt pack, and/or otherancillary devices (e.g., totems, haptic devices, gaming instruments,etc.). Typically, electromagnetic tracking systems include at least oneelectromagnetic field emitter and at least one electromagnetic fieldsensor. Because the emitted electromagnetic fields have a knowndistribution, the detected fields may be analyzed to determine aposition and/or orientation of the peripheral device. Although suchsystems offer a simple solution to the localization problem, there is aneed for additional solutions that offer higher accuracy localization.Embodiments of the present invention can replace or supplementelectromagnetic tracking systems.

Embodiments of the present invention provide a visual tracking systemfor performing high-accuracy localization of a handheld device (e.g., atotem) with respect to a wearable device (e.g., a head set). An imagingdevice is mounted to one of the devices and may capture an image of oneor more fiducials affixed to the other device. An additional imagingdevice may be mounted to the handheld device for capturing variousenvironmental markers. Based on the number of fiducials in the capturedimage, different data processing schemes may be implemented that utilizefiducial data (i.e., data based on the fiducial image having a localreference) and handheld data (data gathered from sensors mounted to thehandheld device having a world reference) differently. Each dataprocessing scheme, referred to herein as an operating state, may enableaccurate estimation of the position and/or orientation of the handhelddevice with respect to the wearable device. The tracking system mayinform the AR system of the estimated localization, and the AR systemmay use the localization information to generate virtual content thatfeels comfortable to the user.

FIG. 1 illustrates an AR scene as viewed through a wearable AR deviceaccording to an embodiment described herein. An AR scene 100 is depictedwherein a user of an AR technology sees a real-world park-like setting106 featuring people, trees, buildings in the background, and a concreteplatform 120. In addition to these items, the user of the AR technologyalso perceives that he “sees” a robot statue 110 standing upon thereal-world platform 120, and a cartoon-like avatar character 102 flyingby, which seems to be a personification of a bumble bee, even thoughthese elements (character 102 and statue 110) do not exist in the realworld. Due to the extreme complexity of the human visual perception andnervous system, it is challenging to produce a VR or AR technology thatfacilitates a comfortable, natural-feeling, rich presentation of virtualimage elements amongst other virtual or real-world imagery elements.

FIG. 2 illustrates various possible components of an AR system. In theillustrated embodiment, an AR system user 260 is depicted wearing a headmounted component 258 featuring a frame 264 structure coupled to adisplay system 262 positioned in front of the eyes of the user. Aspeaker 266 is coupled to frame 264 in the depicted configuration and ispositioned adjacent the ear canal of the user (in one embodiment,another speaker, not shown, is positioned adjacent the other ear canalof the user to provide for stereo/shapeable sound control). Display 262is operatively coupled (as indicated by 268), such as by a wired lead orwireless connectivity, to a local processing and data module 270 whichmay be mounted in a variety of configurations, such as fixedly attachedto frame 264, fixedly attached to a helmet or hat, removably attached tothe torso of user 260 in a backpack-style configuration, or removablyattached to the hip of user 260 in a belt-coupling style configuration.

Local processing and data module 270 may comprise a power-efficientprocessor or controller, as well as digital memory, such as flashmemory, both of which may be utilized to assist in the processing,caching, and storage of data a) captured from sensors which may beoperatively coupled to frame 264, such as image capture devices (such ascameras), microphones, inertial measurement units, accelerometers,compasses, GPS units, radio devices, and/or gyroscopes; and/or b)acquired and/or processed using remote processing module 272 and/orremote data repository 274, possibly for passage to display 262 aftersuch processing or retrieval.

Local processing and data module 270 may be operatively coupled (asindicated by 276, 278), such as via wired or wireless communicationlinks, to remote processing module 272 and remote data repository 274such that these remote modules 272, 274 are operatively coupled to eachother and available as resources to local processing and data module270. In one embodiment, remote processing module 272 may comprise one ormore relatively powerful processors or controllers configured to analyzeand process data and/or image information. In one embodiment, remotedata repository 274 may comprise a relatively large-scale digital datastorage facility, which may be available through the internet or othernetworking configuration in a “cloud” resource configuration. In oneembodiment, all data is stored and all computation is performed in thelocal processing and data module, allowing fully autonomous use from anyremote modules.

FIG. 3 illustrates an example of how a visual tracking system may beincorporated into an AR system having a wearable device 302 (e.g., ahead set) and a handheld device 304 (e.g., a controller). In someembodiments, handheld device 304 may be a handheld controller thatallows a user to provide an input to the AR system. For example,handheld device 304 may be a totem to be used in a gaming scenario.Handheld device 304 may be a haptic device and may include one or morehaptic surfaces utilizing a variety of sensor types. During operation ofthe AR system, a user may hold handheld device 304 in his/her left orright hand by actively gripping handheld device 304 and/or by securingan attachment mechanism (e.g., a wraparound strap) to the user's hand.

Handheld device 304 may include one or more fiducials (referred toherein as handheld fiducials 322) positioned along one or more exteriorsurfaces of handheld device 304 such that the fiducials may be withinthe field of view of an imaging device external to handheld device 304.Handheld fiducials 322 may have a known relationship with respect toeach other such that an imaging device may determine its position and/ororientation with respect to handheld device 304 by capturing an image ofone or more of handheld fiducials 322. Handheld fiducials 322 may bedynamic, static, electrically powered, unpowered, and may, in someembodiments, be distinguishable from each other. For example, a firstfiducial may be a light-emitting diode (LED) having a first wavelengthand a second fiducial may be an LED having a second wavelength.Alternatively or additionally, different fiducials may have differentbrightness and/or may pulsate at different frequencies (e.g., a firstfiducial may pulsate at 100 Hz and a second fiducial may pulsate at 150Hz).

Handheld device 304 may include one or more imaging devices (referred toherein as handheld imaging devices 326) positioned in a manner such thatwearable device 302 and/or some feature in the surroundings of handhelddevice 304 is within the field of view(s) of the imaging device(s) whenhandheld device 304 is being held by a user. For example, a fronthandheld imaging device 326A may be positioned such that its field ofview is oriented away from the user towards one or more features in thesurroundings of handheld device 304, and a rear handheld imaging device326B may be positioned such that its field of view is oriented towardswearable device 302. Handheld imaging devices 326 may include one ormore front-facing imaging devices and/or one or more rear-facing imagingdevices to create a desired cumulative field of view. In someembodiments, handheld imaging devices 326 may be optical devices such ascameras and may capture still or moving images.

Handheld device 304 may include an inertial measurement unit (IMU)(referred to herein as handheld IMU 324) that is rigidly secured withinhandheld device 304 such that rotational and linear movement of handhelddevice 304 is similarly experienced by handheld IMU 324. In someinstances, handheld IMU 324 may include one or more accelerometers(e.g., three), one or more gyroscopes (e.g., three), one or moremagnetometers (e.g., three), and/or digital signal processing hardwareand software to convert raw measurements into processed data. Forexample, handheld IMU 324 may include an accelerometer, a gyroscope, anda magnetometer for each of three axes. For each axis, handheld IMU 324may output one or more of: linear position, linear velocity, linearacceleration, rotational position, rotational velocity, and/orrotational acceleration. Alternatively or additionally, handheld IMU 324may output raw data from which any of the above-mentioned forms ofprocessed data may be calculated.

Handheld device 304 may comprise a rechargeable and/or replaceablebattery 328 or other power supply that powers handheld fiducials 322,handheld imaging devices 326, handheld IMU 324, and any other componentsof handheld device 304. Although not illustrated in FIG. 3, handhelddevice 304 may include circuitry for enabling wireless communicationwith wearable device 302 and/or belt pack 340. For example, upondetecting or capturing data using handheld imaging devices 326 andhandheld IMU 324, handheld device 304 may transmit raw or processed datato wearable device 302 and/or belt pack 340.

Wearable device 302 may include one or more fiducials (referred toherein as wearable fiducials 306) positioned along one or more exteriorsurfaces of wearable device 306 such that the fiducials may be withinthe field of view of rear handheld imaging device 326B. Wearablefiducials 306 may have a known relationship with respect to each othersuch that an imaging device may determine its position and/ororientation with respect to wearable device 306 by capturing an image ofone or more of wearable fiducials 306. Wearable fiducials 306 may bedynamic, static, electrically powered, unpowered, and may, in someembodiments, be distinguishable from each other. For example, a firstfiducial may be an LED having a first wavelength and a second fiducialmay be an LED having a second wavelength. Alternatively or additionally,different fiducials may have different brightness and/or may pulsate atdifferent frequencies.

Wearable device 302 may include one or more imaging devices (referred toherein as wearable imaging device 310) positioned in a manner such thathandheld device 304 (specifically handheld fiducials 322) is within thefield of view(s) of the imaging device(s) when handheld device 304 isbeing held by a user. For example, one or more wearable imaging devices310 may be positioned front-facing on wearable device 302 above, below,and/or to the side of an optical see-through component of wearabledevice 302. In one embodiment, two wearable imaging devices 310 may bepositioned on opposite sides of the optical see-through component ofwearable device 302. In some embodiments, wearable imaging devices 310may be optical devices such as cameras and may capture still or movingimages.

Wearable device 302 may include an IMU (referred to herein as wearableIMU 308) that is rigidly secured within wearable device 302 such thatrotational and linear movement of wearable device 302 is similarlyexperienced by wearable IMU 308. In some instances, wearable IMU 308 mayinclude one or more accelerometers (e.g., three), one or more gyroscopes(e.g., three), one or more magnetometers (e.g., three), and/or digitalsignal processing hardware and software to convert raw measurements intoprocessed data. For example, wearable IMU 308 may include anaccelerometer, a gyroscope, and a magnetometer for each of three axes.For each axis, wearable IMU 308 may output one or more of: linearposition, linear velocity, linear acceleration, rotational position,rotational velocity, and/or rotational acceleration. Alternatively oradditionally, wearable IMU 308 may output raw data from which any of theabove-mentioned forms of processed data may be calculated.

In some embodiments, the AR system may include a belt pack 340, whichmay include a computing apparatus (e.g., one or more processors and anassociated memory) for performing a localization of handheld device 304with respect to wearable device 302. Alternatively or additionally, thecomputing apparatus may reside in wearable device 302 itself, or evenhandheld device 304. The computing apparatus may receive (via a wiredand/or wireless connection) raw or processed data from each of wearableIMU 308, wearable imaging device 310, handheld IMU 324, and handheldimaging devices 326, and may compute a geospatial position of handhelddevice 304 (with respect to the geospatial position of wearable device302) and an orientation of handheld device 304 (with respect to theorientation of wearable device 302). The computing apparatus may in turncomprise a mapping database 342 (e.g., passable world model, coordinatespace, etc.) to detect pose, to determine the coordinates of realobjects and virtual objects, and may even connect to cloud resources andthe passable world model, in one or more embodiments. In someembodiments, images captured using wearable imaging device 310 and/orhandheld imaging devices 326 may be used to build a passable worldmodel. For example, features may be detected in the captured images, andthe collected data (for example sparse points) may be used for buildingthe passable world model or environmental maps otherwise.

FIG. 4 illustrates a diagram of the localization task, as performed bythe AR system, in which the position and the orientation of handhelddevice 304 are determined with respect to wearable device 302. In theillustrated diagram, wearable device 302 has a geospatial position(“wearable position”) defined as (X_(WP), Y_(WP), Z_(WP)) with respectto a world reference and an orientation (“wearable orientation”) definedas (X_(WO), Y_(WO), Z_(WO)) with respect to a world reference. In someinstances, the geospatial position of wearable device 302 is expressedin longitude, latitude, and elevation values and the orientation ofwearable device 302 is expressed in pitch angle, yaw angle, and rollangle values.

As illustrated, handheld device 304 has a geospatial position (“handheldposition”) defined as (X′_(HP), Y′_(HP), Z′_(HP)) with respect to thegeospatial position of wearable device 302 (X_(WP), Y_(WP), Z_(WP)) andan orientation (“handheld orientation”) defined as (X′_(HO), Y′_(HO),Z′_(HO)) with respect to the orientation of wearable device 302 (X_(WO),Y_(WO), Z_(WO)). In some instances, the geospatial position of handhelddevice 304 is expressed in X, Y, and Z Cartesian values and theorientation of handheld device 304 is expressed in pitch angle, yawangle, and roll angle values. As one specific example, when handhelddevice 304 is being held by a user, the geospatial position of handhelddevice 304 may be equal to (0.7 m, −0.5 m, 0.1 m) and the orientation ofhandheld device 304 may be equal to (10.2°, −46.2°, 15.2°).

FIG. 5 illustrates an example configuration of an AR system 500 in whichwearable device 302 includes one or more wearable fiducials 306 andhandheld device 304 includes one or more rear-facing handheld imagingdevices 326 having a field of view that at least partially and at leasttemporarily includes wearable fiducials 306 while handheld device 304 isbeing held by a user in normal operation. AR system 500 may includeadditional sensors mounted to handheld device 304 such as handheld IMU324. One advantage of such a configuration may be that handheld device304 has all the data needed to perform localization of itself withrespect to wearable device 302, thereby reducing the processing load onwearable device 302. AR system 500 may include additional sensorsmounted to wearable device 302 such as wearable IMU 308.

FIG. 6 illustrates a method 600 of performing localization of handhelddevice 304 with respect to wearable device 302 using AR system 500. Oneor more steps of method 600 may be omitted or may be performed in anorder different than the illustrated embodiment, and one or more stepsof method 600 may be performed at one or more processing apparatuslocated within wearable device 302, handheld device 304, and/or beltpack 340.

At step 602, an image (“fiducial image”) is captured by handheld imagingdevice 326. The fiducial image may contain a number of fiducials ofwearable fiducials 306. For example, if there are three wearablefiducials 306, the fiducial image may be analyzed to determine that itcontains zero, one, two, or three fiducials.

At step 604, a position and/or orientation of handheld device 304 withrespect to wearable device 302 is calculated, for example, based on thefiducial image. For example, the fiducial image may be analyzed todetermine the locations of any fiducials of wearable fiducials 306, andthe position and/or orientation may be determined based on the locationsof the fiducial(s) within the fiducial image as well as the knownphysical relationship between wearable fiducials 306. The positionand/or orientation of handheld device 304 may be used to determine apose of handheld device 304 with respect to wearable device 302. Theoutput of step 604 is referred to as fiducial data 630.

At step 610, data (“IMU data”) indicative of at least rotationalmovement of handheld device 304 with respect to the world (and/or withrespect to wearable device 302) is detected by handheld IMU 324. The IMUdata may include rotational velocities or raw data from which rotationalvelocities may be calculated. In some embodiments, the IMU data is alsoindicative of linear movement of handheld device 304, and may includelinear accelerations or raw data from which linear accelerations may becalculated.

At step 612, the position and/or orientation of handheld device 304 iscalculated based on the IMU data. In some embodiments, the positionand/or orientation of handheld device 304 with respect to the world iscalculated (using previous known and/or estimated orientations withrespect to the world) and/or, in some other embodiments, the positionand/or orientation of handheld device 304 with respect to wearabledevice 302 is calculated (using previous known and/or estimatedorientations with respect to wearable device 302).

At step 614, the position and/or orientation of handheld device 304 withrespect to wearable device 302 is calculated based on fiducial data 630and/or handheld data 632. Fiducial data 630 may include the fiducialimage and/or the position and/or orientation calculations based on thefiducial image performed in step 604. Handheld data 632 may include theIMU data and/or the position and/or orientation calculations based onthe IMU data performed in step 612. The position and orientationcalculation at step 614 may be performed in accordance with one ofvarious operating states based on the number of fiducials found in thefiducial image. Each operating state may treat fiducial data 630 andhandheld data 632 differently and may place greater emphasis on one typeof data with respect to the other type of data. The operating states aredescribed in further detail in reference to FIG. 13.

At step 616, the position and/or orientation of handheld device 304 withrespect to wearable device 302 is outputted, for example, to an externaldevice and/or process for use in operating AR system 500. For example,the position and/or orientation may be outputted to AR system 500 forgenerating and displaying virtual content.

FIG. 7 illustrates an example configuration of an AR system 700 in whichwearable device 302 includes one or more wearable imaging devices 310having a field of view that at least partially and at least temporarilyincludes handheld fiducials 322 while handheld device 304 is being heldby a user in normal operation, and handheld device 304 includes one ormore handheld fiducials 322. AR system 700 may include additionalsensors mounted to handheld device 304 such as handheld IMU 324. Oneadvantage of such a configuration may be the simplicity and low-powerconsumption of handheld device 304. AR system 700 may include additionalsensors mounted to wearable device 302 such as wearable IMU 308.

FIG. 8 illustrates a method 800 of performing localization of handhelddevice 304 with respect to wearable device 302 using AR system 700. Oneor more steps of method 800 may be omitted or may be performed in anorder different than the illustrated embodiment, and one or more stepsof method 800 may be performed at one or more processing apparatuslocated within wearable device 302, handheld device 304, and/or beltpack 340.

At step 802, an image (“fiducial image”) is captured by wearable imagingdevice 310. The fiducial image may contain a number of fiducials ofhandheld fiducials 322. For example, if there are three handheldfiducials 322, the fiducial image may be analyzed to determine that itcontains zero, one, two, or three fiducials.

At step 804, a position and/or orientation of handheld device 304 withrespect to wearable device 302 is calculated, for example, based on thefiducial image. For example, the fiducial image may be analyzed todetermine the locations of any fiducials of handheld fiducials 322, andthe position and/or orientation may be determined based on the locationsof the fiducial(s) within the fiducial image as well as the knownphysical relationship between handheld fiducials 322. The positionand/or orientation of handheld device 304 may be used to determine apose of handheld device 304 with respect to wearable device 302. Theoutput of step 804 is referred to as fiducial data 830.

At step 810, data (“IMU data”) indicative of at least rotationalmovement of handheld device 304 with respect to the world is detected byhandheld IMU 324. The IMU data may include rotational velocities or rawdata from which rotational velocities may be calculated. In someembodiments, the IMU data is also indicative of linear movement ofhandheld device 304, and may include linear accelerations or raw datafrom which linear accelerations may be calculated.

At step 812, the position and/or orientation of handheld device 304 iscalculated based on the IMU data. In some embodiments, the positionand/or orientation of handheld device 304 with respect to the world iscalculated (using previous known and/or estimated orientations withrespect to the world) and/or, in some embodiments, the position and/ororientation of handheld device 304 with respect to wearable device 302is calculated (using previous known and/or estimated orientations withrespect to wearable device 302). The output of step 812 may be referredto as handheld data 832

At step 814, the position and/or orientation of handheld device 304 withrespect to wearable device 302 is calculated based on fiducial data 830and/or handheld data 832. Fiducial data 830 may include the fiducialimage and/or the position and/or orientation calculations based on thefiducial image performed in step 804. Handheld data 832 may include theIMU data and/or the position and/or orientation calculations based onthe IMU data performed in step 812. The position and/or orientationcalculation at step 814 may be performed in accordance with one ofvarious operating states based on the number of fiducials found in thefiducial image. Each operating state may treat fiducial data 830 andhandheld data 832 differently and may place greater emphasis on one typeof data with respect to the other type of data. The operating states aredescribed in further detail in reference to FIG. 13.

At step 816, the position and/or orientation of handheld device 304 withrespect to wearable device 302 is outputted, for example, to an externaldevice and/or process for use in operating AR system 700. For example,the position and/or orientation may be outputted to AR system 700 forgenerating and displaying virtual content.

FIG. 9 illustrates an example configuration of an AR system 900 in whichhandheld device 326 includes front handheld imaging device 326A having afield of view that at least partially and at least temporarily includesone or more surrounding features 344 while handheld device 304 is beingheld by a user and rear handheld imaging device 326B having a field ofview that at least partially and at least temporarily includes one ormore wearable fiducials 306 while handheld device 304 is being held by auser in normal operation. In the example configuration, multiplewearable fiducials 322 are affixed to wearable device 302. AR system 900may include additional sensors mounted to handheld device 304 such ashandheld IMU 324. One advantage of such a configuration may be theincreased accuracy provided by the multiple imaging devices. AR system900 may include additional sensors mounted to wearable device 302 suchas wearable IMU 308.

FIG. 10 illustrates a method 1000 of performing localization of handhelddevice 304 with respect to wearable device 302 using AR system 900. Oneor more steps of method 1000 may be omitted or may be performed in anorder different than the illustrated embodiment, and one or more stepsof method 1000 may be performed at one or more processing apparatuslocated within wearable device 302, handheld device 304, and/or beltpack 340.

At step 1002, an image (“fiducial image”) is captured by rear handheldimaging device 326B. The fiducial image may contain a number offiducials of wearable fiducials 306. For example, if there are threewearable fiducials 306, the fiducial image may be analyzed to determinethat it contains zero, one, two, or three fiducials.

At step 1004, the position and/or orientation of handheld device 304with respect to wearable device 302 is calculated based on, for example,the fiducial image. For example, the fiducial image may be analyzed todetermine the locations of any fiducials of wearable fiducials 306, andthe position and/or orientation may be determined based on the locationsof the fiducial(s) within the fiducial image as well as the knownphysical relationship between wearable fiducials 306. The positionand/or orientation of handheld device 304 may be used to determine apose of handheld device 304 with respect to wearable device 302. Theoutput of step 1004 is referred to as fiducial data 1030.

At step 1006, an image (“world image”) is captured by front handheldimaging device 326A. The world image may contain surrounding features344.

At step 1008, the position and/or orientation of handheld device 304with respect to the world is calculated based on the world image. Insome instances, the world image is compared to previous world images toestimate the movement of handheld device 304 using visual odometrytechniques, which may include performing feature detection in each ofthe world images to establish correspondence between the world images.The movement vector of handheld device 304 that is most consistent withthe movement of the detected features in the world images may then becalculated. The output of step 1008 is referred to as handheld data1032.

At step 1010, data (“IMU data”) indicative of at least rotationalmovement of handheld device 304 with respect to the world is detected byhandheld IMU 324. The IMU data may include rotational velocities or rawdata from which rotational velocities may be calculated. In someembodiments, the IMU data is also indicative of linear movement ofhandheld device 304, and may include linear accelerations or raw datafrom which linear accelerations may be calculated.

At step 1012, the position and/or orientation of handheld device 304 iscalculated based on the IMU data. In some embodiments, the positionand/or orientation of handheld device 304 with respect to the world iscalculated (using previous known and/or estimated orientations withrespect to the world) and/or, in some embodiments, the position and/ororientation of handheld device 304 with respect to wearable device 302is calculated (using previous known and/or estimated orientations withrespect to wearable device 302). The output of step 1012 is referred toas handheld data 1032.

At step 1014, the position and/or orientation of handheld device 304with respect to wearable device 302 is calculated based on fiducial data1030 and/or handheld data 1032. Fiducial data 1030 may include thefiducial image and/or the position and/or orientation calculations basedon the fiducial image performed in step 1004. Handheld data 1032 mayinclude the world image, the position and/or orientation calculationsbased on the world image performed in step 1008, the IMU data, and/orthe position and orientation calculations based on the IMU dataperformed in step 1012. The position and/or orientation calculation atstep 1014 may be performed in accordance with one of various operatingstates based on the number of fiducials found in the fiducial image.Each operating state may treat fiducial data 1030 and handheld data 1032differently and may place greater emphasis on one type of data withrespect to the other type of data. The operating states are described infurther detail in reference to FIG. 13.

At step 1016, the position and/or orientation of handheld device 304with respect to wearable device 302 is outputted, for example, to anexternal device and/or process for use in operating AR system 900. Forexample, the position and/or orientation may be outputted to AR system900 for generating and displaying virtual content.

FIG. 11A illustrates an example configuration of an AR system 1100A inwhich wearable device 302 includes one or more wearable imaging devices310 having a field of view that at least partially and at leasttemporarily includes handheld fiducials 322 while handheld device 304 isbeing held by a user in normal operations, and in which handheld device304 includes one or more handheld imaging devices 326 having a field ofview that at least partially and at least temporarily includes one ormore surrounding features 344 while handheld device 304 is being held bya user in normal operation. In the example configuration illustrated inFIG. 11A, a single handheld fiducial 322 is affixed to handheld device304. AR system 1100 may include additional sensors mounted to handhelddevice 304 such as handheld IMU 324. Advantages of the configurationillustrated in FIG. 11A include the increased accuracy provided by themultiple imaging devices as well as the computational efficiency ofcalculating position and orientation while constrained by a singlefiducial location. AR system 1100A may include additional sensorsmounted to wearable device 302 such as wearable IMU 308.

FIG. 11B illustrates an example configuration of an AR system 1100B inwhich wearable device 302 includes one or more wearable imaging devices310 having a field of view that at least partially and at leasttemporarily includes handheld fiducials 322 while handheld device 304 isbeing held by a user in normal operation, and in which handheld device304 includes one or more handheld imaging devices 326 having a field ofview that at least partially and at least temporarily includes one ormore surrounding features 344 while handheld device 304 is being held bya user in normal operation. In the example configuration illustrated inFIG. 11B, multiple handheld fiducials 322 are affixed to handheld device304. AR system 1100B may include additional sensors mounted to handhelddevice 304 such as handheld IMU 324. Advantages of such a configurationinclude the increased accuracy provided by the multiple imaging devicesas well as the increased robustness by combining fiducial-based trackingwith visual odometry techniques. AR system 1100B may include additionalsensors mounted to wearable device 302 such as a IMU.

FIG. 12 illustrates a method 1200 of performing localization of handhelddevice 304 with respect to wearable device 302 using AR system 1100A ofFIG. 11A or AR system 1100B of FIG. 11B. One or more steps of method1200 may be omitted or may be performed in an order different than theillustrated embodiment, and one or more steps of method 1200 may beperformed at one or more processing apparatus located within wearabledevice 302, handheld device 304, and/or belt pack 340.

At step 1202, an image (“fiducial image”) is captured by wearableimaging device 310. The fiducial image may contain a number of handheldfiducials 322. For example, with respect to FIG. 11A, if there is onehandheld fiducial 322, the fiducial image may be analyzed to determinethat it contains zero or one fiducial. For example, with respect to FIG.11B, if there are three handheld fiducials 322, the fiducial image maybe analyzed to determine that it contains zero, one, two, or threefiducials.

At step 1204, the position and/or orientation of handheld device 304with respect to wearable device 302 is calculated, for example, based onthe fiducial image. For example, with respect to FIG. 11A, the fiducialimage may be analyzed to determine the location of the fiducial, and aconstraint for the position and/or orientation may be determined basedon the location of the fiducial within the fiducial image. For example,with respect to FIG. 11B, the fiducial image may be analyzed todetermine the locations of any fiducials, and the position and/ororientation may be determined based on the locations of the fiducial(s)within the fiducial image as well as the known physical relationshipbetween wearable fiducials 306.

At step 1206, an image (“world image”) is captured by handheld imagingdevice 326. The world image may contain surrounding features 344.

At step 1208, the position and/or orientation of handheld device 304with respect to the world is calculated based on the world image. Insome instances, the world image is compared to previous world images toestimate the movement of handheld device 304 using visual odometrytechniques, which may include performing feature detection in each ofthe world images to establish correspondence between the world images.The movement vector of handheld device 304 that is most consistent withthe movement of the detected features in the world images may then becalculated. The output of step 1208 is referred to as handheld data1232.

At step 1210, data (“IMU data”) indicative of at least rotationalmovement of handheld device 304 with respect to the world is detected byhandheld IMU 324. The IMU data may include rotational velocities or rawdata from which rotational velocities may be calculated. In someembodiments, the IMU data is also indicative of linear movement ofhandheld device 304, and may include linear accelerations or raw datafrom which linear accelerations may be calculated.

At step 1212, the position and/or orientation of handheld device 304 iscalculated based on the IMU data. In some embodiments, the positionand/or orientation of handheld device 304 with respect to the world iscalculated (using previous known and/or estimated orientations withrespect to the world) and/or, in some embodiments, the position and/ororientation of handheld device 304 with respect to wearable device 302is calculated (using known and/or estimated orientations with respect towearable device 302). The output of step 1212 is referred to as handhelddata 1232.

At step 1214, the position and/or orientation of handheld device 304with respect to wearable device 302 is calculated based on fiducial data1230 and/or handheld data 1232. For example, with respect to FIG. 11A,fiducial data 1230 may include the fiducial image and/or the constraintfor the position and/or orientation calculation based on the fiducialimage performed in step 1204. For example, with respect to FIG. 11B,fiducial data 1230 may include the fiducial image and/or the positionand/or orientation calculations based on the fiducial image performed instep 1204. Handheld data 1232 may include the world image, the positionand/or orientation calculations based on the world image performed instep 1208, the IMU data, and/or the position and/or orientationcalculations based on the IMU data performed in step 1212. The positionand/or orientation calculation at step 1214 may be performed inaccordance with one of various operating states based on the number offiducials found in the fiducial image. Each operating state may treatfiducial data 1230 and handheld data 1232 differently and may placegreater emphasis on one type of data with respect to the other type ofdata. The operating states are described in further detail in referenceto FIG. 13.

At step 1216, the position and/or orientation of handheld device 304with respect to wearable device 302 is outputted, for example, to anexternal device and/or process for use in operating AR systems 1100. Forexample, the position and/or orientation may be outputted to AR systems1100 for generating and displaying virtual content.

FIG. 13 illustrates a method 1300 of performing localization of handhelddevice 304 with respect to wearable device 302 using any one of ARsystems 500, 700, 900, 1100 or any combination thereof. One or moresteps of method 1300 may be omitted or may be performed in an orderdifferent than the illustrated embodiment, and one or more steps ofmethod 1300 may be performed at one or more processing apparatus locatedwithin wearable device 302, handheld device 304, and/or belt pack 340.

At step 1302, data (“fiducial data”) indicative of movement of handhelddevice 304 with respect to wearable device 302 is obtained using animaging device. Performing step 1302 may including performing one orboth of steps 1304, 1306. At step 1304, an image (“fiducial image”)containing a number of wearable fiducials 306 is captured by rearhandheld imaging device 326B. At step 1306, an image (“fiducial image”)containing a number of handheld fiducials 322 is captured by wearableimaging device 310.

At step 1308, data (“handheld data”) indicative of at least rotationalmovement of handheld device 304 with respect to the world is detected.Performing step 1308 may include performing one or both of steps 1310,1312.

At step 1310, an image (“world image”) is captured by front handheldimaging device 326A containing surrounding features 344. At step 1312,data (“IMU data”) indicative of at least rotational movement of handhelddevice 304 with respect to the world is detected by handheld IMU 324.The IMU data may include rotational velocities or raw data from whichrotational velocities may be calculated. In some embodiments, the IMUdata is also indicative of linear movement of handheld device 304, andmay include linear accelerations or raw data from which linearaccelerations may be calculated.

At step 1314, the number of fiducials contained in the fiducial image isdetermined as well as the locations (e.g., pixel locations) of theobserved fiducials.

At step 1316, the position and/or orientation of handheld device 304with respect to wearable device 302 is calculated/estimated/updated inaccordance with one of three operating states. An operating state isselected based on the number of fiducials that are observed in thefiducial image. In the illustrated embodiment, the first operating state(“State 1”) is selected when three or more fiducials are observed in thefiducial image, the second operating state (“State 2”) is selected whenone or two fiducials are observed in the fiducial image, and the thirdoperating state (“State 3”) is selected when zero fiducials are observedin the fiducial image. Switching between states may occur each time anew fiducial image is captured or at predetermined intervals. Forexample, step 1316 may be performed at each camera frame based on one orboth of the fiducial data (e.g., the fiducial image) and the handhelddata (e.g., the world image and the IMU orientation). Step 1316 mayfurther incorporate previous position and/or orientation calculations toimprove estimation accuracy.

In accordance with the first operating state (“State 1”), the positionand/or orientation may be calculated (in full six degrees of freedom)with high accuracy, for example, based solely on the fiducial data. Whenfour or more fiducials are observed, the position can be completelysolved for. When exactly three fiducials are observed, two possiblesolutions to the position exist, one of which can be discarded based onadditional processing and/or comparisons to previously calculatedpositions. In some embodiments, the handheld data may be used tosupplement and improve the calculation accuracy. In some embodiments, anextended Kalman filter may be employed to improve accuracy based onprevious position and/or orientation calculations.

In accordance with the second operating state (“State 2”), the positionand/or orientation may be calculated, for example, using both thefiducial data and the handheld data. When two fiducials are observed,the fiducial data enables a constrained position and/or orientation tobe calculated, and the handheld data may be used to complete thecalculation under the constraint imposed by the fiducial data. In someembodiments, an extended Kalman filter may be employed to improveaccuracy based on previous position and/or orientation calculations.Calculations performed under the second operating state may overall beless accurate than calculations performed under the first operatingstate.

In accordance with the third operating state (“State 3”), the positionand orientation may be calculated, for example, based solely on thehandheld data (i.e., dead reckoning). In some embodiments, an extendedKalman filter may be employed to improve accuracy based on previousposition and/or orientation calculations. Calculations performed underthe third operating state may overall be less accurate than calculationsperformed under the first or second operating states.

At step 1318, IMU bias corrections are performed to increase theaccuracy of the IMU data provided as inputs at step 1316. Because theIMU data may drift over time, periodic updates can recalibrate the IMUdata. In some embodiments, bias updates are only provided when the firstoperating state is selected and high-accuracy bias updates can beprovided. In some embodiments, bias updates are provided when either thefirst operating state or the second operating state is selected, as bothstates utilize fiducial data in their calculations. Bias updates can beprovided at each camera frame or at predetermined intervals.

FIG. 14 illustrates a simplified computer system 1400 according to someembodiments described herein. FIG. 14 provides a schematic illustrationof one example of computer system 1400 that can perform some or all ofthe steps of the methods provided by various embodiments. It should benoted that FIG. 14 is meant only to provide a generalized illustrationof various components, any or all of which may be utilized asappropriate. FIG. 14, therefore, broadly illustrates how individualsystem elements may be implemented in a relatively separated orrelatively more integrated manner.

Computer system 1400 is shown comprising hardware elements that can beelectrically coupled via a bus 1405, or may otherwise be incommunication, as appropriate. The hardware elements may include one ormore processors 1410, including without limitation one or moregeneral-purpose processors and/or one or more special-purpose processorssuch as digital signal processing chips, graphics accelerationprocessors, and/or the like; one or more input devices 1415, which caninclude without limitation a mouse, a keyboard, a camera, and/or thelike; and one or more output devices 1420, which can include withoutlimitation a display device, a printer, and/or the like.

Computer system 1400 may further include and/or be in communication withone or more non-transitory storage devices 1425, which can comprise,without limitation, local and/or network accessible storage, and/or caninclude, without limitation, a disk drive, a drive array, an opticalstorage device, a solid-state storage device, such as a random accessmemory (“RAM”), and/or a read-only memory (“ROM”), which can beprogrammable, flash-updateable, and/or the like. Such storage devicesmay be configured to implement any appropriate data stores, includingwithout limitation, various file systems, database structures, and/orthe like.

Computer system 1400 might also include a communications subsystem 1419,which can include without limitation a modem, a network card (wirelessor wired), an infrared communication device, a wireless communicationdevice, and/or a chipset such as a Bluetooth™ device, an 802.11 device,a WiFi device, a WiMax device, cellular communication facilities, etc.,and/or the like. The communications subsystem 1419 may include one ormore input and/or output communication interfaces to permit data to beexchanged with a network such as the network described below to name oneexample, other computer systems, television, and/or any other devicesdescribed herein. Depending on the desired functionality and/or otherimplementation concerns, a portable electronic device or similar devicemay communicate image and/or other information via the communicationssubsystem 1419. In other embodiments, a portable electronic device, e.g.the first electronic device, may be incorporated into computer system1400, e.g., an electronic device as an input device 1415. In someembodiments, computer system 1400 will further comprise a working memory1435, which can include a RAM or ROM device, as described above.

Computer system 1400 also can include software elements, shown as beingcurrently located within the working memory 1435, including an operatingsystem 1440, device drivers, executable libraries, and/or other code,such as one or more application programs 1445, which may comprisecomputer programs provided by various embodiments, and/or may bedesigned to implement methods, and/or configure systems, provided byother embodiments, as described herein. Merely by way of example, one ormore procedures described with respect to the methods discussed above,might be implemented as code and/or instructions executable by acomputer and/or a processor within a computer; in an aspect, then, suchcode and/or instructions can be used to configure and/or adapt a generalpurpose computer or other device to perform one or more operations inaccordance with the described methods.

A set of these instructions and/or code may be stored on anon-transitory computer-readable storage medium, such as the storagedevice(s) 1425 described above. In some cases, the storage medium mightbe incorporated within a computer system, such as computer system 1400.In other embodiments, the storage medium might be separate from acomputer system e.g., a removable medium, such as a compact disc, and/orprovided in an installation package, such that the storage medium can beused to program, configure, and/or adapt a general purpose computer withthe instructions/code stored thereon. These instructions might take theform of executable code, which is executable by computer system 1400and/or might take the form of source and/or installable code, which,upon compilation and/or installation on computer system 1400 e.g., usingany of a variety of generally available compilers, installationprograms, compression/decompression utilities, etc., then takes the formof executable code.

It will be apparent to those skilled in the art that substantialvariations may be made in accordance with specific requirements. Forexample, customized hardware might also be used, and/or particularelements might be implemented in hardware, software including portablesoftware, such as applets, etc., or both. Further, connection to othercomputing devices such as network input/output devices may be employed.

As mentioned above, in one aspect, some embodiments may employ acomputer system such as computer system 1400 to perform methods inaccordance with various embodiments of the technology. According to aset of embodiments, some or all of the procedures of such methods areperformed by computer system 1400 in response to processor 1410executing one or more sequences of one or more instructions, which mightbe incorporated into the operating system 1440 and/or other code, suchas an application program 1445, contained in the working memory 1435.Such instructions may be read into the working memory 1435 from anothercomputer-readable medium, such as one or more of the storage device(s)1425. Merely by way of example, execution of the sequences ofinstructions contained in the working memory 1435 might cause theprocessor(s) 1410 to perform one or more procedures of the methodsdescribed herein. Additionally or alternatively, portions of the methodsdescribed herein may be executed through specialized hardware.

The terms “machine-readable medium” and “computer-readable medium,” asused herein, refer to any medium that participates in providing datathat causes a machine to operate in a specific fashion. In embodimentsimplemented using computer system 1400, various computer-readable mediamight be involved in providing instructions/code to processor(s) 1410for execution and/or might be used to store and/or carry suchinstructions/code. In many implementations, a computer-readable mediumis a physical and/or tangible storage medium. Such a medium may take theform of a non-volatile media or volatile media. Non-volatile mediainclude, for example, optical and/or magnetic disks, such as the storagedevice(s) 1425. Volatile media include, without limitation, dynamicmemory, such as the working memory 1435.

Common forms of physical and/or tangible computer-readable mediainclude, for example, a floppy disk, a flexible disk, hard disk,magnetic tape, or any other magnetic medium, a CD-ROM, any other opticalmedium, punchcards, papertape, any other physical medium with patternsof holes, a RAM, a PROM, EPROM, a FLASH-EPROM, any other memory chip orcartridge, or any other medium from which a computer can readinstructions and/or code.

Various forms of computer-readable media may be involved in carrying oneor more sequences of one or more instructions to the processor(s) 1410for execution. Merely by way of example, the instructions may initiallybe carried on a magnetic disk and/or optical disc of a remote computer.A remote computer might load the instructions into its dynamic memoryand send the instructions as signals over a transmission medium to bereceived and/or executed by computer system 1400.

The communications subsystem 1419 and/or components thereof generallywill receive signals, and the bus 1405 then might carry the signalsand/or the data, instructions, etc. carried by the signals to theworking memory 1435, from which the processor(s) 1410 retrieves andexecutes the instructions. The instructions received by the workingmemory 1435 may optionally be stored on a non-transitory storage device1425 either before or after execution by the processor(s) 1410.

The methods, systems, and devices discussed above are examples. Variousconfigurations may omit, substitute, or add various procedures orcomponents as appropriate. For instance, in alternative configurations,the methods may be performed in an order different from that described,and/or various stages may be added, omitted, and/or combined. Also,features described with respect to certain configurations may becombined in various other configurations. Different aspects and elementsof the configurations may be combined in a similar manner. Also,technology evolves and, thus, many of the elements are examples and donot limit the scope of the disclosure or claims.

Specific details are given in the description to provide a thoroughunderstanding of exemplary configurations including implementations.However, configurations may be practiced without these specific details.For example, well-known circuits, processes, algorithms, structures, andtechniques have been shown without unnecessary detail in order to avoidobscuring the configurations. This description provides exampleconfigurations only, and does not limit the scope, applicability, orconfigurations of the claims. Rather, the preceding description of theconfigurations will provide those skilled in the art with an enablingdescription for implementing described techniques. Various changes maybe made in the function and arrangement of elements without departingfrom the spirit or scope of the disclosure.

Also, configurations may be described as a process which is depicted asa schematic flowchart or block diagram. Although each may describe theoperations as a sequential process, many of the operations can beperformed in parallel or concurrently. In addition, the order of theoperations may be rearranged. A process may have additional steps notincluded in the figure. Furthermore, examples of the methods may beimplemented by hardware, software, firmware, middleware, microcode,hardware description languages, or any combination thereof. Whenimplemented in software, firmware, middleware, or microcode, the programcode or code segments to perform the necessary tasks may be stored in anon-transitory computer-readable medium such as a storage medium.Processors may perform the described tasks.

Having described several example configurations, various modifications,alternative constructions, and equivalents may be used without departingfrom the spirit of the disclosure. For example, the above elements maybe components of a larger system, wherein other rules may takeprecedence over or otherwise modify the application of the technology.Also, a number of steps may be undertaken before, during, or after theabove elements are considered. Accordingly, the above description doesnot bind the scope of the claims.

As used herein and in the appended claims, the singular forms “a”, “an”,and “the” include plural references unless the context clearly dictatesotherwise. Thus, for example, reference to “a user” includes a pluralityof such users, and reference to “the processor” includes reference toone or more processors and equivalents thereof known to those skilled inthe art, and so forth.

Also, the words “comprise”, “comprising”, “contains”, “containing”,“include”, “including”, and “includes”, when used in this specificationand in the following claims, are intended to specify the presence ofstated features, integers, components, or steps, but they do notpreclude the presence or addition of one or more other features,integers, components, steps, acts, or groups.

It is also understood that the examples and embodiments described hereinare for illustrative purposes only and that various modifications orchanges in light thereof will be suggested to persons skilled in the artand are to be included within the spirit and purview of this applicationand scope of the appended claims.

1. (canceled)
 2. A method of performing localization of a handhelddevice with respect to a wearable device, the method comprising:obtaining, by a sensor mounted on the handheld device, handheld dataindicative of movement of the handheld device; obtaining, by an imagingdevice mounted on the wearable device, fiducial data indicative ofmovement of the handheld device, wherein obtaining the fiducial dataincludes capturing, by the imaging device, a fiducial image containing anumber of fiducials affixed to the handheld device; determining thenumber of fiducials contained in the fiducial image; and updating aposition and an orientation of the handheld device using the fiducialdata and the handheld data and based on the number of fiducialscontained in the fiducial image.
 3. The method of claim 2 wherein thesensor comprises an inertial measurement unit (IMU).
 4. The method ofclaim 2 wherein updating the position and orientation of the handhelddevice is accordance with a first operating state or a second operatingstate.
 5. The method of claim 4, further comprising: in response todetermining that the number of fiducials is equal to or greater thanthree, updating at least one of the position and the orientation of thehandheld device based on the fiducial data in accordance with the firstoperating state; and in response to determining that the number offiducials is equal to one or two, updating at least one of the positionand the orientation of the handheld device based on the fiducial dataand the handheld data in accordance with the second operating state. 6.The method of claim 5, further comprising in response to determiningthat the number of fiducials is equal to zero, updating at least one ofthe position and the orientation of the handheld device based on thehandheld data in accordance with a third operating state.
 7. The methodof claim 6 wherein: at least one of the position and the orientation ofthe handheld device is updated based solely on the fiducial data inaccordance with the first operating state; and at least one of theposition and the orientation of the handheld device is updated basedsolely on the handheld data in accordance with the third operatingstate.
 8. The method of claim 2 wherein the number of fiducialscomprises a number of light-emitted diodes (LEDs).
 9. The method ofclaim 2 wherein: the sensor comprises a second imaging device; andobtaining the handheld data includes capturing, by the second imagingdevice, a world image containing one or more features surrounding thehandheld device.
 10. A method of performing localization of a handhelddevice with respect to a wearable device, the method comprising:obtaining handheld data from a sensor mounted on the handheld device;obtaining fiducial data from an imaging device mounted on either thewearable device or the handheld device, wherein the fiducial dataincludes a fiducial image containing a number of fiducials affixed toeither the wearable device or the handheld device; determining thenumber of fiducials contained in the fiducial image; and determiningwhether the number of fiducials contained in the fiducial image is equalto one of a first set of values or to one of a second set of valuesdifferent from the first set of values; in response to determining thatthe number of fiducials contained in the fiducial image is equal to oneof the first set of values, updating a position and an orientation ofthe handheld device based on the fiducial data or the fiducial data andthe handheld data in accordance with a first operating state; and inresponse to determining that the number of fiducials contained in thefiducial image is equal to one of the second set of values, updating theposition and the orientation of the handheld device based on thefiducial data and the handheld data or the handheld data in accordancewith a second operating state different from the first operating state.11. The method of claim 10 wherein the first set of values includesintegers equal to or greater than three.
 12. The method of claim 11wherein the position and orientation is updated based solely on thefiducial data.
 13. The method of claim 10 wherein the second set ofvalues consists of integers equal to one or two.
 14. The method of claim10 wherein the second set of values is equal to zero, the method furthercomprising in response to determining that the number of fiducialscontained in the fiducial image is equal to zero, updating the positionand the orientation of the handheld device based solely on the handhelddata.
 15. The method of claim 10 wherein the number of fiducialscomprises a number of light-emitted diodes (LEDs).
 16. The method ofclaim 10 wherein the imaging device is mounted on the handheld deviceand a plurality of fiducials including the number of fiducials areaffixed to the wearable device.
 17. The method of claim 16 whereinobtaining the handheld data includes capturing, by a second handheldimaging device mounted to the handheld device, a world image containingone or more features surrounding the handheld device.
 18. The method ofclaim 10 wherein the imaging device is mounted on the wearable deviceand a plurality of fiducials including the number of fiducials areaffixed to the handheld device.
 19. The method of claim 18 wherein thesensor comprises an inertial measurement unit (IMU).
 20. The method ofclaim 18 wherein obtaining the handheld data includes capturing, by asecond handheld imaging device mounted to the handheld device, a worldimage containing one or more features surrounding the handheld device.21. The method of claim 10 wherein the sensor comprises an inertialmeasurement unit (IMU).